Cryptographic key distribution

ABSTRACT

An HSM management hub coordinates the distribution and synchronization of cryptographic material across a fleet of connected hardware security modules (“HSMs”). Cryptographic material is exchanged between HSMs in the fleet in a cryptographically protected format. In some examples, the cryptographic material is encrypted using a common fleet key maintained by the HSMs in the fleet. In other examples, the cryptographic material is protected using asymmetric cryptographic keys that are associated with the members of the HSM fleet. The HSM management hub may be used to divide the HSM fleet into subdomains by providing domain keys to subsets of HSMs within the HSM fleet. Cryptographic information that is encrypted with particular domain keys can be distributed across the entire HSM fleet, and restricted to use by authorized HSMs that are in possession of the particular domain keys.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/958,808, filed Dec. 3, 2015, entitled “CRYPTOGRAPHIC KEYDISTRIBUTION,” the full disclosure of which is incorporated by referencefor all purposes herein. This application also incorporates by referencefor all purposes the full disclosure of co-pending U.S. patentapplication Ser. No. 14/958,816, filed Dec. 3, 2015, entitled “STORAGEOF CRYPTOGRAPHIC INFORMATION.”

BACKGROUND

Maintaining the security of computer systems is a difficult problem. Oneway that computer systems are secured, is through the use ofcryptography. Cryptographic systems rely on one or more cryptographickeys to protect information. In many situations the security of thecryptographic keys is paramount, because access to the cryptographickeys allows access to the data that is being protected. One waycryptographic keys can be secured is through the use of a HardwareSecurity Module (“HSM”). An HSM is a physical computing device thatsafeguards cryptographic keys by storing them within a tamper-resistantphysical device. HSMs provide cryptographic key generation and storage,and perform cryptographic operations for authorized clients of the HSM.Some cryptographic keys, called session keys, are associated with aparticular connection from a particular HSM client, and are deleted as aresult of termination of the session. Token keys are cryptographic keysthat persist on the HSM, and that can be used by multiple users andsessions. In general, the cryptographic keys are not exportable from theHSM in an unprotected form.

In large-scale computing environments, the demand for cryptographicoperations may exceed the capabilities of a single HSM. To improve theperformance of cryptographic operations, some HSMs provide accelerationof cryptographic operations. In distributed computing environments,multiple HSMs may be used to provide cryptographic functions to variousservers and clients throughout the distributed environment. Maintaininga fleet of HSMs can be difficult, because the non-exportability of thecryptographic keys makes it difficult to maintain a collection ofsynchronized cryptographic keys across the fleet.

BRIEF DESCRIPTION OF THE DRAWINGS

Various techniques will be described with reference to the drawings, inwhich:

FIG. 1 shows an illustrative example of an environment in which variousembodiments may be practiced;

FIG. 2 shows an illustrative example of an HSM with fleet-managementfeatures, in accordance with an embodiment;

FIG. 3 shows an illustrative example of an HSM management hub configuredto manage a number of HSMs as a fleet, in accordance with an embodiment;

FIG. 4 shows an illustrative example of an HSM management hubestablishing an HSM domain that includes two HSMs, in accordance with anembodiment;

FIG. 5 shows an illustrative example of a process that, as a result ofbeing performed by an HSM management hub, a first HSM in the fleet, anda second HSM in the fleet, establishes an HSM domain that includes thefirst and second HSMs, in accordance with an embodiment;

FIG. 6 shows an illustrative example of an HSM management hubsynchronizing a domain key across a number of HSMs in a fleet, inaccordance with an embodiment;

FIG. 7 shows an illustrative example of a process that, as a result ofbeing performed by an HSM management hub, a first member HSM, a secondmember HSM, and a non-member HSM, synchronizes an application key acrossa number of HSMs in a fleet, in accordance with an embodiment;

FIG. 8 shows an illustrative example of two HSMs synchronizing a fleetkey, in accordance with an embodiment;

FIG. 9 shows an illustrative example of a process that, as a result ofbeing performed by a first HSM and a second HSM, copies a fleet key fromthe first HSM to the second HSM, in accordance with an embodiment;

FIG. 10 shows an illustrative example of an HSM management hubdistributing fleet-configuration information and cryptographic materialto a fleet of HSMs through a hierarchical distribution network, inaccordance with an embodiment;

FIG. 11 shows an illustrative example of an HSM management hubdistributing fleet-configuration information to a fleet of HSMs througha hierarchical distribution network, and the fleet of HSMs distributingcryptographic material throughout the fleet using peer-to-peerdistribution, in accordance with an embodiment;

FIG. 12 shows an illustrative example of an HSM management hubparticipating in peer-to-peer distribution of network of cryptographicmaterial throughout a fleet of HSMs, in accordance with an embodiment;

FIG. 13 shows an illustrative example of a storage service used forretaining protected cryptographic material maintained by a fleet ofHSMs, in accordance with an embodiment;

FIG. 14 shows an illustrative example of cryptographic materialdistributed across a fleet of customer HSMs, with associated parityinformation maintained by an HSM management hub, in accordance with anembodiment;

FIG. 15 shows an illustrative example of a storage service used toretain parity information for cryptographic information maintained by afleet of HSMs, in accordance with an embodiment;

FIG. 16 shows an illustrative example of a process that, as a result ofbeing performed by an HSM fleet and a storage service, backs upinformation stored on the HSM fleet by exporting parity information to astorage service, in accordance with an embodiment;

FIG. 17 shows an illustrative example of a process that, as a result ofbeing performed by an HSM fleet and a storage service, restores a failedHSM using parity information retained by the storage service, inaccordance with an embodiment; and

FIG. 18 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

The current document describes systems and methods that distribute andsynchronize cryptographic information across a fleet of HSMs. A numberof HSMs are organized into a fleet that is managed by an HSM managementhub. Each HSM in the fleet is provided with a copy of a domain key forthe fleet called a fleet key. A fleet management agent is installed oneach HSM that communicates with the HSM management hub and facilitatesoperations which are coordinated by the HSM management hub. The HSMmanagement hub communicates with the members of the fleet, andfacilitates the exchange and synchronization of cryptographicinformation between the HSMs. The HSM management hub is able tofacilitate the formation of HSM domains, distribute updatedcryptographic information, and backup and restore HSM data.

The fleet can include various types of HSMs such as internal HSMs,directly attached HSMs, and network HSMs. An internal HSM is an HSM thatis contained within a host computer system such as a PC, laptop, ortablet computer. Internal HSMs may take the form of an expansion cardsuch as a PCI card, PCMCIA card, or an SD card that is installed in thehost computer system. A directly attached HSM is an HSM peripheral thathas a direct physical connection to a host computer. Examples ofdirectly attached HSMs include HSMs that are connected to the hostcomputer with a USB connection, a serial connection, or a FireWireconnection. A network HSM is an HSM that is connected to a computernetwork. Clients of a network HSM connect to the network HSM over thecomputer network, and in some examples, multiple clients are able toaccess a single HSM.

The HSM management hub is a computing appliance that operates an HSMfleet manager service and maintains a database of fleet information. TheHSM fleet manager service includes a configuration manager and a fleetdata manager. The configuration manager manages the structure of the HSMfleet by recording information such as fleet membership, domainmembership, and HSM hierarchies. The fleet data manager managesinformation stored on the fleet of HSMs by propagating and synchronizingcryptographic information across the fleet of HSMs in accordance withthe structure of the fleet. In general, the cryptographic informationpropagated by the HSM management hub is protected by cryptographic keysmaintained on the HSMs to which the cryptographic information ispropagated. For example, an application key generated on a first HSM isencrypted with the fleet key on the first HSM. The encrypted applicationkey can be provided to the HSM management hub and replicated to one ormore additional fleet member HSMs. The additional fleet member HSMs areable to decrypt and use the application key using their duplicates ofthe fleet key. The HSM management hub does not have access to the fleetkey, and therefore cannot decrypt or use the application key.

A fleet key can be installed on a new fleet-member HSM in a variety ofways. In some examples, the fleet key is installed by the manufacturerof the HSM at the time the HSM is made. In another example, the fleetkey is installed on HSM using one or more pieces of physical computerreadable media. The fleet key is divided into a number of segments andeach segment is placed onto a separate piece of physical computerreadable storage media such as an SD card, CDROM, or USB drive. Thephysical computer readable media are inserted into the new HSM, and thenew HSM assembles and installs the fleet key.

Cryptographic information may be distributed across a number of HSMsusing public-private key cryptography. In one example, a fleet key isinstalled on a new HSM using a public-private key pair generated by thenew HSM. The new HSM provides the public key to a fleet member HSM, andthe fleet member HSM encrypts the fleet key with the public key providedby the new HSM. The fleet member HSM provides the encrypted fleet key tothe new HSM, and the encrypted fleet key is decrypted with the privatekey on the new HSM. In this way, the fleet key is not exposed in anunencrypted form outside the set of HSM in the fleet. In anotherexample, an HSM management hub coordinates the distribution of the fleetkey. The HSM management hub contacts the HSMs in the fleet, and causeseach HSM in the fleet to generate an HSM-specific public-private keypair. Each HSM in the fleet provides the public key of the HSM-specificpublic-private key pair to the HSM management hub. The HSM managementhub provides the collection of public keys to a fleet member HSM. Thefleet member HSM generates the fleet key, and encrypts the fleet keywith each public key provided by the HSM management hub. The collectionof encrypted the keys is distributed via the HSM management hub to themembers of the fleet. Each member of the fleet decrypts the fleet keywith the member's HSM-specific private key.

HSM management functions can be implemented in a distributed fashion bythe member HSMs. Communication between HSMs may be facilitated by acommonly accessible storage service, or by a peer-to-peer data sharingservice implemented by the HSMs. In some examples, HSMs within a fleetconnect to a network storage service using a shared storage-serviceaccount. HSMs in the fleet exchange shared information with the storageservice in a protected format using a shared fleet key. The shared fleetkey may be established across the fleet of HSMs using public-private keyencryption as described above. In another example, HSMs within the fleetcommunicate with each in a peer-to-peer fashion other usingpublic-private key encryption. Each HSM in the fleet communicates with anumber of peer fleet member HSMs for which the HSM is provided withpublic keys.

By providing an HSM management hub that manages cryptographicinformation across a fleet of HSMs, persistent cryptographic objectscreated on one HSM can be automatically replicated to other HSMs in thefleet. If the HSM management hub fails, each HSM in the fleet may beable to continue functioning using locally-stored cryptographicinformation stored within each HSM. A secure communications channel maybe created that relies on the replication of cryptographic information.For example, a secure network connection may be established between twonetwork-connected hosts using a shared secret key. Each host isconnected to an HSM that is a member of the HSM fleet. The shared secretkey is generated on one HSM and replicated across the fleet of HSMs,providing the two network-connected hosts access to the shared secretkey.

In some implementations, cryptographic information is exchanged betweenHSMs within the HSM fleet in a hierarchical or peer-to-peer fashion, anddomain and fleet keys are distributed directly from an HSM managementhub. By controlling the distribution of domain keys centrally, accurateaccess control can be maintained. Cryptographic information such ascryptographic keys are encrypted with one or more domain keys or thefleet key and distributed relatively freely between HSMs within the HSMfleet. Only HSMs that have the corresponding domain or fleet keys mayaccess the cryptographic information.

Cryptographic information can be backed up using stored services outsidethe HSM fleet. In some examples, backup cryptographic information isgenerated on HSM and encrypted using a fleet key. The encrypted backupinformation is transmitted to a storage service where it is maintainedin encrypted form. In another example, backup cryptographic informationis generated on HSM and split into a number of backup stripes which aredistributed across HSMs in the HSM fleet. Parity information isgenerated that enables recovery from the loss of one or more backupstripes. The parity information may be maintained on an HSM managementhub, or an external storage service. If an HSM failure occurs within anHSM fleet, the parity information can be recovered and used toregenerate the lost backup stripe.

FIG. 1 shows an illustrative example of an environment in which variousembodiments may be practiced. A diagram 100 includes an HSM managementhub that manages a number of HSMs that are members of an HSM fleet. TheHSM fleet includes a first HSM 104, a second HSM 106, and a third HSM108. The first HSM 104, the second HSM 106, and the third HSM 108 retaincryptographic information 112, 114, and 116 respectively which issynchronized by the HSM management hub 102 across the fleet. Whencryptographic information is exchanged between the HSMs 104, 106, and108, and the HSM management hub 102, the cryptographic information iscryptographically protected using a fleet key 118, 120, and 122, that issynchronized across the HSM fleet.

The fleet key is a cryptographic key used to protect the cryptographicinformation when the cryptographic information is transmitted outsidethe HSMs in the fleet. In some examples, the fleet key is a symmetriccryptographic fleet key. Cryptographic information can be encryptedusing the symmetric cryptographic fleet key and transmitted from asource fleet-member HSM. After receipt by a recipient fleet-member HSM,the recipient can decrypt the protected cryptographic information usingthe symmetric cryptographic fleet key and merge the receivedcryptographic information with local cryptographic information stored onthe recipient fleet-member HSM. In other examples, the fleet key is anasymmetric cryptographic fleet key comprising a public fleet key and aprivate fleet key. A source fleet-member HSM can encrypt cryptographicinformation with the public fleet key and send the encryptedcryptographic information to a destination fleet-member HSM. Thedestination fleet-member HSM is able to decrypt the cryptographicinformation using the private fleet key, and then synchronize thereceived cryptographic information with local cryptographic informationstored on the destination fleet-member HSM. Fleet-member HSMs areconfigured so that that the fleet key is not usable by HSM clients. Ingeneral, the fleet key is usable only by a fleet management agent withinthe HSM itself.

The fleet key may be created by the HSM management hub when the fleet iscreated. The HSM management hub creates the fleet by identifying aninitial fleet-member HSM. The HSM management hub generates anadministrative key that is used to sign fleet commands issued by the HSMmanagement hub to fleet-member HSMs. The administrative key is anasymmetric key that includes a public key and a private key. The privatekey is maintained by the HSM management hub. The public key isdistributed to the fleet-member HSMs to validate commands issued by theHSM management hub. In order to generate the initial fleet key, the HSMmanagement hub provides the initial fleet-member HSM with the publickey, and issues a signed command to generate a new fleet key.

The new fleet key can be distributed to new HSMs in the fleet withoutrevealing the fleet key to the HSM management hub. In one example, theHSM management hub contacts a new fleet-member HSM, and requests thatthe new fleet-member HSM join the fleet. The HSM management hub providesthe public administrative key to the new fleet-member HSM. The newfleet-member HSM generates a fleet-import public-private key pair foruse in receiving the new fleet key. The new fleet-member HSM providesthe fleet-import public key to the HSM management hub. The newfleet-member HSM may sign the fleet-import public key with an HSMmanufacturer's key to indicate that the fleet-import private key resideswithin a secure HSM. The HSM management hub signs the fleet-importpublic key with the administrative key and provides the fleet-importpublic key to a fleet-member HSM that has fleet key. The fleet-memberHSM encrypts the fleet key with the provided fleet-import public keyafter validating the HSM manufacturer's key and the administrative keysignatures. The encrypted fleet key is returned to the new fleet-memberHSM via the HSM management hub. The encrypted fleet key is decryptedusing the fleet-import private key and installed on the new fleet-memberHSM.

Communication between HSMs may occur in a variety of ways. In someimplementations, communication between HSMs occurs via the HSMmanagement hub 102. The HSM management hub 102 includes an HSM fleetmanager 124 that interfaces with a fleet information store 126. Thefleet information store 126 includes information that describes how tocommunicate with each member of the HSM fleet, information thatdescribes how the HSM fleet is arranged, and encrypted copies ofcryptographic information maintained by the fleet. The HSM fleet manager124 periodically contacts each member of the HSM fleet and receivesupdated cryptographic material from each member of the HSM fleet. Theupdated cryptographic material is distributed to the remaining membersof the HSM fleet. The cryptographic material is cryptographicallyprotected with the fleet key before the cryptographic material leaveseach HSM, and the cryptographic protection is removed after thecryptographic information arrives at each destination HSM using thefleet key. In another implementation, fleet-member HSMs communicate witheach other in a peer-to-peer fashion. Cryptographic information may betransferred between HSMs in the fleet in a cryptographically protectedformat that is encrypted with the fleet key or, as an alternative, thecryptographic information may be exchanged between fleet-member HSMsusing a secure transport protocol such as TLS that relies on anegotiated shared secret specific to the connection.

Updated cryptographic information may be generated on one HSM andsynchronized across the fleet of HSMs to support requests forcryptographic operations from HSM clients. In one example, a clientcomputer system is connected to the first HSM 104. The client computersystem requests generation of a new cryptographic key for use by anapplication. The new cryptographic key is generated on the first HSM104, and an identifier associated with the new cryptographic key isprovided to the application. In general, the first HSM 104 does notprovide the new cryptographic key in plaintext form to the application.The application requests that the HSM perform cryptographic operationsusing the new cryptographic key. In this way, the application may usethe new cryptographic key to cryptographically protect application datathat is stored or shared with other applications without accessing thenew cryptographic key directly. If the protected application data isshared with other applications, and the other applications do not haveaccess to the first HSM 104, the other applications may use other HSMsin the HSM fleet to decrypt the protected application data if the newcryptographic key is synchronized across the HSM fleet. If the clientcomputer system requests that the new cryptographic key be synchronizedacross the HSM fleet, the first HSM 104 cryptographically protects thenew application key with the fleet key, and then transmits the protectedfleet key to the HSM management hub 102. The HSM management hub 102transmits the protected new application key to the other HSMs in thefleet. The other HSMs in the fleet decrypt the protected new applicationkey using the fleet key, and retain the new application key for use byHSM clients.

The HSM management hub 102 may be used to perform key rotationoperations that are synchronized across the fleet of HSMs. In oneexample of key rotation, the HSM management hub 102 indicates to thefirst HSM 104 that the fleet key should be rotated. The first HSM 104generates a new fleet key and cryptographically protects the new fleetkey with the old fleet key. The cryptographically protected new fleetkey is provided to the HSM management hub 102. The HSM management hub102 provides the protected new fleet key to the remaining HSMs whichdecrypt and retain the new fleet key. If an HSM is unreachable, the HSMmanagement hub 102 retains the protected new fleet key and can providethe protected new fleet key to the unreachable HSM if the unreachableHSM becomes available at a later time. In another example of keyrotation, the HSM management hub 102 indicates to the first HSM 104 thatan application key should be rotated. The first HSM 104 generates a newapplication key, and cryptographically protects the new application keywith the original application key and the fleet key. The protected newapplication key is transmitted to the HSM management hub 102 anddistributed to the remaining HSMs. Only HSMs that have access to boththe original application key and the fleet key can access the newapplication key. If, for example, an HSM were stolen or compromised, theHSM management hub 102 could quickly rotate the fleet key of theremaining HSMs, and then rotate the remaining application and domainkeys across the fleet using the new fleet key.

The HSM management hub 102 may retain protected cryptographicinformation in the fleet information store 126 for backup and recoverypurposes. In one example, the HSM fleet manager 124 periodicallyrequests a backup from an HSM in the fleet. The HSM cryptographicallyprotects the cryptographic information stored on the HSM using the fleetkey and provides the protected cryptographic information to the HSMfleet manager 124. The HSM fleet manager 124 stores the cryptographicinformation in the fleet information store 126 in protected form. TheHSM management hub 102 cannot use the cryptographic information in thefleet information store 126 without access to the fleet key. In general,the fleet key is not usable outside the HSM fleet because the fleet keyis non-exportable, and not usable by HSM clients. In a restoreoperation, the protected cryptographic information may be provided to anHSM fleet member, provided the HSM fleet member has already beenprovided with the fleet key.

If the fleet key needs to be restored as well, the target HSM generatesa public-private key pair and provides the public key to a fleet memberHSM that has the fleet key. The public key provided by the target HSMmay be signed with an HSM manufacturer's key and a fleet administrator'skey to indicate that the associated private key is securely retainedwithin an HSM and authorized to receive the fleet key. The fleet memberHSM encrypts the fleet key with the provided public key and returns theencrypted fleet key to the target HSM. The target HSM decrypts theencrypted fleet key with the private key and stores the resulting fleetkey in the target HSM.

FIG. 2 shows an illustrative example of an HSM with fleet-managementfeatures, in accordance with an embodiment. A diagram 200 shows an HSM202 that is connected to a network 204 via a client computer system 203.In some examples, the HSM 202 may be a directly connected HSM connectedto the client computer system 203 via a USB or serial connection. Inanother example, the HSM 202 may be an internal HSM installed within theclient computer system 203 such as a PCI card, a PCMCIA card, or an SDcard. The client computer system 203 can be a personal computer, alaptop computer, a tablet computer, a mobile device, or cell phone thatuses the HSM 202 to manage cryptographic keys and/or performcryptographic operations.

The HSM 202 includes a fleet management agent 206. The fleet managementagent 206 is able to connect to an HSM management hub via the network204 by relaying information through the client computer system 203. Insome implementations, the client computer system 203 includes a driverthat acts as a communication relay between the fleet management agent206 and a network interface on the client computer system 203. Thedriver relays messages received from the HSM management hub via thenetwork interface to the fleet management agent 206, and relays messagestransmitted by the fleet management agent 206 to the HSM management hubvia the network interface on the client computer system 203.

The HSM 202 includes a crypto processor 208, and in some embodiments, acryptographic accelerator 210. The crypto processor 208 is atamper-resistant processor that carries out cryptographic operations ina secure environment. The crypto processor 208 does not revealinstructions or data on an external accessible bus in an unencryptedform. If an attempt is made to breach the physical security of the HSM202, the crypto processor 208 will delete the data maintained within thecrypto processor. Some HSMs include a cryptographic accelerator 210. Thecryptographic accelerator 210 is a coprocessor that is able to performcomputationally intensive cryptographic operations. The cryptographicaccelerator 210 can offload cryptographic operations from the cryptoprocessor 208 and, in some implementations, the client computer system203.

The HSM 202 maintains a key store 212 that contains cryptographicmaterial used by the HSM 202 and the client computer system 203. The keystore 212 includes an application key 214, a domain key 216, and a fleetkey 218. The application key 214 is a cryptographic key used by anapplication on the client computer system 203. The application key 214is generated on the HSM 202 at the request of the application.Cryptographic operations may be requested by the application andperformed by the HSM 202 using the application key 214, but theapplication key 214 is not exposed outside the HSM 202. The domain key216 is a symmetric cryptographic key that is shared across to a set ofHSMs within a larger fleet of HSMs. The domain key 216 is replicatedacross the set of HSMs. Cryptographic material such as the applicationkey 214 may be assigned to the domain and encrypted with a domain key216. The encrypted cryptographic material is replicated across the setof HSMs in the domain to provide the set of HSMs with access to thecryptographic material. In some implementations, the encryptedcryptographic material is replicated across the entire fleet of HSMs,but only the HSMs in the set of HSMs are able to access thecryptographic material because only the HSMs in the set of HSMs haveaccess to the domain key 216. The fleet key 218 is a symmetriccryptographic key shared across a domain of HSMs managed by an HSMmanagement hub. The fleet key 218 and the domain key 216 may be usedonly within the HSM 202 by the crypto processor 208 or the cryptographicaccelerator 210. With a domain key 216 or the fleet key 218, the HSM 202may encrypt cryptographic material from within the HSM 202 and exportthe encrypted cryptographic material outside the HSM 202, or the HSM 202may import encrypted cryptographic material from outside the HSM 202 anddecrypt the encrypted cryptographic material for use within the HSM 202.

The fleet key 218 may be established across the fleet of HSMs in anumber of ways. In some implementations, the fleet key 218 ispreinstalled by the manufacturer of the HSM 202. While the HSM 202 isbeing manufactured, the fleet key 218 can be written directly to the keystore 212 using methods that are inaccessible after the manufacturingprocess is complete. In another implementation, the fleet key 218 isinstalled using physical media that is inserted directly into the HSM202. The fleet key 218 is divided into a number of segments which aredistributed across a number of pieces of physical media. The pieces ofphysical media are inserted directly into the HSM 202, and the HSM 202imports the fleet key into the key store 212. In yet anotherimplementation, the fleet key 218 is generated by an HSM within thefleet and distributed to the remaining fleet members by encrypting thefleet key with an asymmetric cryptographic key generated by each of theremaining fleet members. For example, the fleet key may be transmittedfrom a source HSM to the destination HSM by generating a public-privatekey pair on the destination HSM, providing the public key to the sourceHSM, and encrypting the fleet key on the source HSM using the publickey, and then sending the encrypted fleet key from the source HSM to thedestination HSM. The destination HSM decrypts the encrypted fleet keywith the private key. In yet another implementation, the fleet key maybe transmitted from one HSM in the fleet to another HSM in the fleetusing a secure transport protocol such as TLS that negotiates sharedsecret using a shared secret negotiation protocol such as theDiffie-Hellman key exchange algorithm.

FIG. 3 shows an illustrative example of an HSM management hub configuredto manage a number of HSMs as a fleet, in accordance with an embodiment.A diagram 300 includes an HSM management hub 302 that manages a firstHSM 304, a second HSM 306, and a third HSM 308 through a network 310.The first HSM 304 is a directly connected HSM that is connected to aclient computer system 311 via a USB or serial connection. The first HSM304 communicates with the HSM management hub 302 through the clientcomputer system 311. The client computer system 311 includes a driverthat relays communications between the first HSM 304 and the HSMmanagement hub 302. The second HSM 306 is a network connected HSM, andcommunicates with the HSM management hub 302 over the network 310. Thethird HSM 308 is an internal HSM that is installed within an HSM hostcomputer system 312. The third HSM 308 may be a PCI card, or otherinternal peripheral that is compatible with the HSM host computer system312. The third HSM 308 communicates with the HSM management hub 302 byrelaying messages through the HSM host computer system 312. The HSM hostcomputer system 312 may include a driver that relays messages betweenthe third HSM 308 and the HSM fleet manager 318 a network 310. The firstHSM 304, the second HSM 306, and the third HSM 308 are members of an HSMfleet, and each member of the HSM fleet has a copy of a fleet key 313,314, and 316. Fleet-member HSMs may be network-connected HSMs that areconnected directly to the network 310, or internal or direct-connectedHSMs that are connected indirectly to the network 310 through a clientcomputer system.

The HSM management hub 302 is a computing appliance or general purposecomputer system that manages the fleet of HSMs. The HSM management hub302 includes an HSM fleet manager 318. The HSM fleet manager 318 is anapplication that runs on the HSM management hub 302. The HSM fleetmanager 318 has subcomponents that include a configuration manager 320and a fleet data manager 322. The configuration manager 320 manages thehierarchy of the HSMs contained in the HSM fleet. For example, theconfiguration manager maintains information about the HSMs that belongto the HSM fleet such as the network address and port for communicatingwith each HSM in the HSM fleet, as well as domain membershipinformation. The configuration manager 320 maintains the informationabout the HSMs in a fleet configuration store 324. Informationmaintained in the fleet configurations for 324 may be cryptographicallyprotected using an administrative key maintained by the HSM managementhub 302. The fleet data manager 322 manages the cryptographic materialsynchronized between HSMs in the HSM fleet. The cryptographic materialmay be stored temporarily or persistently in a cryptographic materialstore 326. The cryptographic material may include fleet keys, domainkeys, and application keys that are synchronized between HSMs in the HSMfleet. The cryptographic material maintained in the cryptographicmaterial store 326 is maintained in a cryptographically protected formatgenerated by the HSMs in the HSM fleet. In general, cryptographicmaterial exchanged between the HSMs is protected using the fleet key,and in some situations, additional cryptographic keys maintained withinthe HSMs. Cryptographic material exchanged between the HSMs may beadditionally protected using an HSM manufacturer's key, a domain key, oran administrative key maintained by the HSM management hub.

An administrative console 328 is used to perform administrativefunctions with the HSM management hub 302. The administrative console328 communicates with the HSM fleet manager 318 and allows anadministrator to add or remove HSMs from the HSM fleet, arrange HSMsinto domains within the HSM fleet, configure HSM backups, and configurefleet synchronization parameters. For example, the administrator canconfigure how often HSMs within the HSM fleet are polled for updatedcryptographic material by the HSM management hub 302. The administratorcan associate particular cryptographic material, such as an applicationkey, with a particular domain. By associating particular cryptographicmaterial with a particular domain, the cryptographic material issynchronized across the HSMs that are members of the domain. If theparticular cryptographic material is associated with the HSM fleet, thecryptographic material is synchronized across the entire HSM fleet. Ifthe particular cryptographic material is not associated with anyparticular domain or with the HSM fleet, the cryptographic material isnot synchronized across any other HSMs.

FIG. 4 shows an illustrative example of an HSM management hubestablishing an HSM domain that includes two HSMs, in accordance with anembodiment. A diagram 400 shows an HSM management hub 402 copying adomain key from a first HSM 404 to a second HSM 406 over a computernetwork 408. The first HSM 404 and the second HSM 406 may be directlyconnected to the computer network 408 or may be indirectly connected tothe computer network 408 via an HSM client. The first HSM 404 includes afleet management agent 410, a cryptoprocessor 412, a cryptographicaccelerator 414, and a cryptographic material store 416. Thecryptographic material store 416 includes a fleet key 418 and a domainkey 420. The second HSM 406 includes a fleet management agent 422, acryptoprocessor 424, a cryptographic accelerator 426, and acryptographic material store 428. The cryptographic material store 428includes a fleet key 430 that is a duplicate of the fleet key 418. Adomain key 432 is an old version of the domain key 420 or, in somescenarios may not yet be present on the cryptographic material store428. The HSM management hub 402 includes an HSM fleet manager 434 and afleet information store 436.

The HSM management hub 402 synchronizes the domain key 420 on the firstHSM 404 with the domain key 432 on the second HSM 406. The HSMmanagement hub 402 detects that the first HSM 404 contains updatedcryptographic information. In some implementations, the HSM fleetmanager 434 contacts the fleet management agent 410 and inquires whetherthe first HSM 404 contains updated cryptographic information. The fleetmanagement agent 410 determines whether the cryptographic material store416 contains updated cryptographic information. If the cryptographicmaterial store 416 contains updated cryptographic information, the fleetmanagement agent 410 encrypts the updated cryptographic informationusing the fleet key 418 and returns the encrypted updated cryptographicinformation to the HSM fleet manager 434. In another implementation, thefleet management agent 410 detects when cryptographic material on thefirst HSM 404 has been updated. When cryptographic material on the firstHSM 404 is updated, the fleet management agent 410 cryptographicallyprotects the updated cryptographic material with the fleet key 418 andsends the protected updated cryptographic material to the HSM fleetmanager 434. When the HSM fleet manager 434 receives updatedcryptographic information, the HSM fleet manager 434 distributes theupdated cryptographic information to other HSMs in the HSM fleet. Insome implementations, the HSM fleet manager 434 distributes updatedcryptographic information to all HSMs in the HSM fleet. In anotherimplementation, the HSM fleet manager 434 distributes the cryptographicinformation based at least in part on domain membership of thecryptographic information.

Cryptographic information that is associated with a domain key can becryptographically protected with the domain key in addition to beingcryptographically protected with the fleet key before being distributedby an HSM. The HSM management hub may distribute the cryptographicinformation across the entire HSM fleet, but only HSMs with a domain keywill be able to access the cryptographic information. In someimplementations, the cryptographic information is protected with adomain key alone.

Communications between the HSM management hub 402 and the HSMs may besigned with an administrative key by the HSM management hub. Theadministrator key is a public-private key pair generated by the HSMmanagement hub at the time the HSM fleet is formed. The public key ofthe administrative key is provided to the fleet-member HSMs. HSMs thatreceive messages from the HSM management hub can confirm theauthenticity of the message by validating the message using theadministrative key.

FIG. 5 shows an illustrative example of a process that, as a result ofbeing performed by an HSM management hub, a first HSM in the fleet, anda second HSM in the fleet, establishes an HSM domain that includes thefirst and second HSMs, in accordance with an embodiment. A processdiagram 500 shows a process that begins at block 502 with the HSMmanagement hub identifying a set of HSMs to be added to a proposeddomain. The set of HSMs are selected from fleet-member HSMs, so thateach HSM in the proposed domain of HSMs has a copy of a fleet key. Atblock 504, the HSM management hub selects an HSM from the proposeddomain of HSMs to generate the domain key, and sends the selected HSM acommand to generate the domain key. The command to generate the domainkey may be signed by the HSM management hub with an administrative keyto indicate to the selected HSM that the command is authorized by afleet administrator.

At block 506, the selected HSM receives the command to generate thedomain key, and generates the domain key. In general, the domain key isa symmetric cryptographic key, but in some implementations, the domainkey is an asymmetric cryptographic key pair. At block 508, the selectedHSM uses the fleet key to cryptographically protect the newly generateddomain key. The newly generated domain key may be cryptographicallyprotected using a variety of encryption algorithm such as the advancedencryption standard (“AES”), data encryption standard (“DES”), blowfish,or CAST, which is described in RFC 2144, RFC 2612, and RFC 2984, whichare hereby incorporated by reference. At block 510, the selected HSMjoins the new domain by recording the new domain key. The new domain keyis returned 512 to the HSM management hub, in a format that is encryptedwith the fleet key.

At block 514, the HSM management hub stores the encrypted domain key.HSM management hub identifies the remaining members of the proposeddomain of HSMs, and synchronizes 516 the encrypted domain key with themembers of the proposed domain. The HSM management hub sends theencrypted domain key to a member of the proposed domain. At block 518,the member of the proposed domain unpacks the encrypted domain key usingthe fleet key to gain access to the domain key and use form. At block520, the member of the proposed domain joins the new domain by recordingthe new domain key.

Domain keys and fleet keys allow HSMs to limit the distribution ofcryptographic material across to the fleet of HSMs. By using a fleet keyor a domain key, domains of HSMs can synchronize cryptographicallyprotected information through the HSM management hub using a key that isnot known outside the domain of HSMs. HSMs outside the domain of HSMs ornon-HSM-based storage services may be used to store cryptographicallyprotected information for backup purposes, and the cryptographicallyprotected information cannot be used unless it is returned within thecryptographic domain of the keys used to protect the information.

FIG. 6 shows an illustrative example of an HSM management hubsynchronizing a domain key across a number of HSMs in a fleet, inaccordance with an embodiment. A diagram 600 includes an HSM managementhub 602 managing a fleet of three HSMs: a first HSM 604, a second HSM606, and a third HSM 608. The HSMs in the fleet can be directlyconnected HSMs, internal HSMs are network connected HSMs. The HSMmanagement hub 602 includes an HSM fleet manager 610 and a cryptographicinformation store 612. The HSM fleet manager 610 synchronizescryptographic material across the fleet of HSMs and the cryptographicinformation store 612 maintains hierarchical information describing thefleet structure and maintains cryptographic material in a protectedformat.

The HSM management hub 602 synchronizes an application key across thethree HSMs in the fleet. The application key is associated with adomain, and the first HSM 604 and the second HSM 606 are members of thedomain. The first HSM 604 retains the current application key 614. Thefirst HSM 604 also includes a domain key 616 and a fleet key 618. Thesecond HSM 606 includes a stale copy of the application key 620. Thesecond HSM 606 also includes a copy of the domain key 622 and a copy ofthe fleet key 624. The third HSM 608 does not have access to the domainkey, but does have a second copy of the fleet key 628.

The application key is replicated across the fleet as follows. The firstHSM 604 encrypts the current application key 614 using the domain key616. The encrypted application key is transmitted to the HSM managementhub 602 and received by the HSM fleet manager 610. In someimplementations, the HSM fleet manager stores the encrypted applicationkey in the cryptographic information store 612. The HSM fleet manageridentifies the remaining members of the fleet and sends the encryptedapplication key to the remaining members. When the second HSM 606receives the encrypted application key, the second HSM 606 decrypts theencrypted application key using the copy of the domain key 622, and thedecrypted application key is used to update the stale copy of theapplication key 620. When the third HSM 608 receives the encryptedapplication key, third HSM 608 retains the encrypted application key 626on the third HSM 608 in encrypted form.

The encrypted application key 626 may be used to recover the applicationkey if all HSMs that are members of the domain become available. In thecase of disaster recovery, the HSM fleet manager 610 accesses theprotected domain key from the cryptographic information store 612 andprovides the protected domain key to the replacement HSMs. Thereplacement HSMs use the fleet key to restore the domain key. The HSMfleet manager extracts the encrypted application key 626 from the thirdHSM 608 and distributes the encrypted application key 626 thereplacement HSMs. The replacement HSMs extract the application key withthe restored domain key, returning the fleet to its previous state.

FIG. 7 shows an illustrative example of a process that, as a result ofbeing performed by an HSM management hub, a first member HSM, a secondmember HSM, and a non-member HSM, synchronizes an application key acrossa number of HSMs in a fleet, in accordance with an embodiment. A processdiagram 700 illustrates a process that begins at block 702 where an HSMmanagement hub receives a request to generate a new application key. Insome situations, the request to generate a new application key mayoriginate with an administrator and be received by the HSM managementhub via an administrative console. In other situations, the request togenerate the new application key may originate with an applicationhosted by a computer system that is connected to an HSM in the HSMfleet. The HSM management hub generates a command 704 to generate a newapplication key associated with the domain, and sends the command to afirst domain member HSM. In some implementations, the command is signedwith an administrative key owned by the HSM management hub.

At block 706, the first domain member HSM receives the command togenerate a new application key. The first domain member HSM generatesthe application key using random or pseudorandom number generationtechniques. At block 708 the first domain member HSM cryptographicallyprotects the application key using the domain key stored on the firstdomain member HSM. The application key may be cryptographicallyprotected using various encryption techniques described elsewhere inthis application. At block 710, the first domain member HSM returns theprotected application key to the HSM management hub.

HSM management hub identifies 712 the destination HSMs for the newlygenerated application key. In some examples, the newly generatedapplication key is distributed across the entire fleet of HSMs, and HSMsthat are able to decrypt the newly generated application key install theapplication key for use by HSM clients. Applications that are not ableto decrypt the newly generated application key may retain theapplication key as a backup for the HSM fleet, or may delete theunusable cryptographic information. At block 714 the HSM management hubdistributes the protected application key to the destination HSMs.

A second domain member HSM receives the protected application key fromthe HSM management hub and unpacks 716 the protected application keyusing the domain key. At block 718, the second domain member HSM is ableto access the application key in a usable format. At block 720, anon-member HSM receives the protected application key from the HSMmanagement hub. The non-member HSM does not have access to the domainkey, and therefore is unable to extract the application key into ausable format. At block 722, the non-member HSM retains the applicationkey in protected format as a backup for the domain member HSMs.

FIG. 8 shows an illustrative example of two HSMs synchronizing a fleetkey, in accordance with an embodiment. A diagram 800 shows a first HSM802 and a second HSM 804. The first HSM 802 retains a cryptographic datastore 806. The cryptographic data store 806 includes a public-privatecryptographic key pair associated with the first HSM 802. Thepublic-private cryptographic key pair includes a first HSM private key808 and a first HSM public key 810. The first HSM 802 maintains acurrent fleet key 812. The second HSM 804 does not have a copy of thecurrent fleet key. The second HSM 804 includes a cryptographic datastore 814. The cryptographic data store 814 maintains a public-privatecryptographic key pair associated with the second HSM 804. Thepublic-private cryptographic key pair includes a second HSM private key816 and a second HSM public key 818. An out-of-date fleet key 820 may bepresent on the cryptographic data store 814. In some scenarios, thesecond HSM 804 may not have any fleet key.

The current fleet key 812 may be replicated from the first HSM 802 tothe second HSM 804 without exposing the fleet key in an unprotectedformat outside the HSMs. The second HSM 804 provides the second HSMpublic key 818 to the first HSM 802. The first HSM 802 cryptographicallyprotects the current fleet key 812 with the received second HSM publickey 818. The first HSM 802 transmits the protected current fleet key 812to the second HSM 804. The second HSM 804 receives the protected currentfleet key, and decrypts the protected current fleet key using the secondHSM private key 816. The decrypted fleet key is used to update theout-of-date fleet key 820. In some embodiments, keys may be transmittedindirectly through an HSM management hub. The HSM management hub can beused to sign and authenticate requests and exchanges of cryptographicinformation between HSMs.

FIG. 9 shows an illustrative example of a process that, as a result ofbeing performed by a first HSM and a second HSM, copies a fleet key fromthe first HSM to the second HSM, in accordance with an embodiment. Aprocess diagram 900 illustrates a process that begins at block 902 wherea second HSM requests a copy of a fleet key from a first HSM. The firstHSM receives 904 the request for the fleet key from the second HSM. Thefirst HSM authorizes the request at block 906. The request may beauthorized by verifying a signature on the request or by confirming therequest with an HSM management hub that manages the fleet. Identifyinginformation on the second HSM may be compared to information maintainedby the HSM management hub to confirm that the second HSM is anauthorized fleet member.

The fleet key is provided from the first HSM to the second HSM in aprotected format so that the fleet key is not revealed outside thecontext of the HSMs. At block 908, the first HSM requests a public keyfrom the second HSM. The second HSM generates a public-private key pairand provides 910 the public key to the first HSM. The first HSM receivesthe public key from the second HSM and then encrypts 912 the currentfleet key with the received public key. The encrypted current fleet keyis transmitted 914 to the second HSM. The second HSM receives theencrypted current fleet key, and then decrypts 916 the encrypted currentfleet key with the private key of the public-private key pair. At block918, the second HSM stores the current fleet key and becomes a member ofthe HSM fleet.

In some embodiments, multiple hardware security modules can be used tosecurely provide a set of keys from one hardware security module to theother. Each security module may be provided with a common domain key. Ahardware security module may encrypt cryptographic material using thedomain key and provide the encrypted cryptographic material to anotherhardware security module, which can use its own copy of the domain keyto decrypt the cryptographic material. A domain key may be provided to ahardware security module using secure transfer methods, such aspublic-key key exchange algorithms. Various other security measures,such as conditions on quorums of operators of security modules may alsobe utilized in accordance with the various embodiments. Additionalinformation may be found in U.S. patent application Ser. No. 13/765,283,filed Feb. 12, 2013, which is incorporated herein by reference.

FIG. 10 shows an illustrative example of an HSM management hubdistributing fleet-configuration information and cryptographic materialto a fleet of HSMs through a hierarchical distribution network, inaccordance with an embodiment. A diagram 1000 shows a system thatincludes an HSM management hub 1002 that is in communication with afleet of HSMs. The HSM management hub 1002 includes an HSM fleet manager1004. The HSM fleet manager 1004 includes a number of softwarecomponents, such as a configuration manager 1006 and a fleet datamanager 1008, that implement management functions supported by the HSMmanagement hub 1002. The configuration manager 1006 maintains a fleetconfiguration data store 1010. The fleet configuration data store 1010includes information that identifies the HSMs in the fleet of HSMsmanaged by the HSM management hub 1002. The fleet configuration datastore 1010 may include information that describes how the HSM managementhub 1002 communicates with each HSM, as well as the role of each HSMwithin the fleet. For example, the fleet configuration data stored 1010may include information that identifies particular subgroups of HSMswithin the HSM fleet. The fleet data manager 1008 maintains informationin a cryptographic material store 1012. The cryptographic material store1012 retains cryptographic keys and other sensitive material provided bythe HSMs in the HSM fleet. Cryptographic material maintained in thecryptographic material store 1012 is maintained in a protected format.In some examples, the cryptographic material is encrypted with a fleetkey.

The fleet of HSMs is arranged in a hierarchy. The HSM management hub1002 distributes fleet configuration information as well as updates tocryptographic material to HSMs within the HSM fleet through a number ofintermediate HSMs. The fleet of HSMs includes a first tier of HSMs and asecond tier of HSMs. The first tier of HSMs includes a first HSM 1014and a second HSM 1016. The second tier of HSMs includes a third HSM1018, a fourth HSM 1020, and a fifth HSM 1022 arranged hierarchicallyunder the first HSM 1014. A sixth HSM 1024, a seventh HSM 1026, and aneighth HSM 1028 are arranged under the second HSM 1016. By arranging theHSM in a hierarchy, fleet configuration information and cryptographicmaterial may be quickly distributed from the HSM management hub 1002 tothe members of the HSM fleet. The HSM management hub distributesinformation to the first HSM 1014 and the second HSM 1016. The first HSMdistributes the received information to the third HSM 1018, the fourthHSM 1020, and fifth HSM 1022. The second HSM distributes information tothe sixth HSM 1024, the seventh HSM 1026, and the eighth HSM 1028. TheHSM fleet may be arranged to provide desired fan-out characteristicswhen distributing information from the HSM management hub 1002.

The hierarchy of HSMs may be arranged based at least in part on theinterconnections of individual HSMs within the HSM fleet. In someexamples, a number of network HSMs may be connected to a common networksegment. A particular HSM selected from the number of network HSMs isselected to act as a master, and receives updates from the HSMmanagement hub 1002. As a result of receiving updates from the HSMmanagement hub 1002, the master HSM distributes the updates to theremaining HSMs that are connected to the network segment. In anotherexample, a host computer system acts as a master node distributinginformation to a number of connected HSMs such as PCI-connected internalHSMs or USB-connected external HSMs connected to the host computersystem. The host computer system includes a driver that is incommunication with the HSM management hub 1002. The driver receivesupdated fleet configuration information and/or cryptographic materialfrom the HSM management hub 1002, and relays the updates to HSMsconnected to the host computer system. By distributing information in ahierarchical fashion, updates can be distributed rapidly across thefleet of HSMs. In some implementations, individual HSMs or the HSMmanagement hub 1002 can employ broadcast functionality to distributeprotected fleet configuration or cryptographic material to a number ofHSMs simultaneously.

FIG. 11 shows an illustrative example of an HSM management hubdistributing fleet-configuration information to a fleet of HSMs througha hierarchical distribution network, and the fleet of HSMs distributingcryptographic material throughout the fleet using peer-to-peerdistribution, in accordance with an embodiment. A system diagram 1100includes an HSM management hub 1102 that manages a fleet of HSMs. Thefleet of HSMs includes a first HSM 1104, a second HSM 1106, and a thirdHSM 1108. The HSM management hub 1102 is a computing appliance thatincludes an HSM fleet manager 1110. The HSM fleet manager 1110 is anexecutable component of the HSM management hub that maintains anddistributes fleet information stored on a fleet information store 1112.The fleet information includes information that controls access tocryptographic keys maintained within the fleet of HSMs. The fleetinformation store 1112 includes domain keys which are encrypted with afleet key accessible to the HSMs within the HSM fleet. The HSMmanagement hub distributes the encrypted domain keys to selected HSMswithin the HSM fleet. The selected HSMs form HSM groups that are enabledto use cryptographic information that is encrypted with particulardomain keys. As a result of being encrypted with a domain key,cryptographic information such as application keys can be distributedfreely between HSMs within the HSM fleet, and access to thecryptographic information is limited to those HSMs that have access tothe appropriate domain keys and/or fleet key.

In the example illustrated in FIG. 11, a number of domain keys aredistributed by the HSM fleet manager 1110 to the HSMs in the HSM fleet.The first HSM 1104, the second HSM 1106, and the third HSM 1108 eachretain a copy of the fleet key 1114, 1116, and 1118 respectively. As aresult of configuring the fleet of HSMs, the HSM fleet manager 1110distributes a number of domain keys to selected HSMs within the HSMfleet. In the illustrated example, the HSM fleet manager 1110distributes a shared domain key to the first HSM 1104 and the third HSM1108. A first domain key 1120 is generated by the first HSM 1104, andencrypted using the fleet key 1114. The encrypted first domain key istransmitted to the HSM fleet manager 1110 and relayed to the third HSM1108. The third HSM 1108 decrypts the encrypted first domain key andretains a copy of the first domain key 1122. The second HSM 1106 isprovided with a second domain key 1124. Domain keys may be retained bythe HSM management hub 1102 as part of the fleet information store 1112.

Cryptographic information such as encryption keys may be encrypted withparticular domain keys and distributed across the HSM fleet in apeer-to-peer fashion. HSMs that have access to the appropriate domainkey are able to decrypt the cryptographic information for use. HSMs thatdo not have access to the appropriate domain key may retain theencrypted cryptographic information for distribution to other HSMs, butmay not decrypt and use the cryptographic information. The first HSM1104 maintains an application key set 1125. The application key set 1125can include a collection of cryptographic keys. Each cryptographic keyin the collection of cryptographic keys may be encrypted by the fleetkey 1114 and one or more domain keys present on the first HSM 1104. Theapplication key set 1125 may be transmitted via a peer-to-peer protocol,broadcast, or otherwise provided to other HSMs in the fleet. In theexample shown in FIG. 11, the first HSM 1104 transmits the applicationkey set 1125 to the second HSM 1106. The second HSM 1106 transmits acopy of the application key set 1126 to the third HSM 1108. The thirdHSM 1108 stores an additional copy of the application key set 1128. TheHSMs each contain identical application key sets. The first HSM 1104 andthe third HSM 1108 are able to decrypt and use cryptographic keys fromthe application key sets that are encrypted with domain key 1 or thefleet key. The second HSM 1106 can decrypt and use cryptographic keysfrom the application key sets that are encrypted with domain key 2.

Distribution of fleet information such as domain keys is tightlycontrolled by the HSM management hub 1102, but distribution ofcryptographic information encrypted with the domain keys may beexchanged between the HSMs using less controlled means such as networkbroadcast, gossip, or peer-to-peer protocols. This can enable the rapiddistribution and synchronization of cryptographic information across theHSM fleet, while maintaining tightly controlled usage rights to thecryptographic information via the domain keys.

In some implementations cryptographic information is distributedthroughout a fleet of HSMs using a gossip-style protocol. Gossip-styleprotocols are described in pending U.S. patent application Ser. No.14/740,218, filed Jun. 15, 2015, which is incorporated herein byreference.

FIG. 12 shows an illustrative example of an HSM management hubparticipating in peer-to-peer distribution of network of cryptographicmaterial throughout a fleet of HSMs, in accordance with an embodiment. Adiagram 1200 shows a system that includes a variety of HSMs that areinterconnected in an ad hoc peer-to-peer network. An HSM management hub1202 includes an HSM fleet manager component 1204 that manages adatabase of cryptographic material 1206. The cryptographic materialincludes encryption keys that are encoded with fleet or domain keysmaintained by the HSMs in the fleet. The HSMs in the fleet areinterconnected in a variety of ways. In HSM host 1208 is connected tothe HSM management hub 1202 via a direct connection such as a FireWireor USB connection. The HSM host 1208 includes an internal HSM 1210. Theinternal HSM 1210 can be installed in the HSM host 1208 by a PCIinterface, and ISA interface, and EISA interface, a PC card interface,or other internal interface supported by the HSM host 1208. AdditionalHSMs are connected via a network 1212. A first network-connected HSM1214 and a second network-connected HSM 1216 are connected directly tothe network 1212. The first network-connected HSM 1214 and the secondnetwork-connected HSM 1216 can include network interfaces such as10baseT Ethernet, 100baseT Ethernet, Gigabit Ethernet, token ring, FDDI,Wi-Fi, or cellular interfaces. The network router 1218 is connected tothe network 1212 and provides an isolated subnet. A thirdnetwork-connected HSM 1220 is connected via a network interface to theisolated subnet provided by the network router 1218. An HSM clientcomputer 1222 is connected via the subnet to the network router 1218.The HSM client computer 1222 is connected to a directly connected HSM1224. The directly connected HSM 1224 may be connected to the HSM clientcomputer 1222 via a USB, FireWire, serial connection, RS-232 connection,Bluetooth connection, or other dedicated communication link.

Communications between the HSMs are orchestrated in a peer-to-peerfashion. For example, updates to cryptographic information that occur onthe first network-connected HSM 1214 may be broadcast over the network1212 to the second network-connected HSM 1216, the network router 1218,and the HSM management hub 1202. The network router 1218 repeats theupdates to the third network-connected HSM 1220 and directly connectedHSM 1224 via the HSM client computer 1222. The HSM management hub 1202relays the updates to the internal HSM 1210 via the HSM host 1208. Insome implementations, the fleet of HSMs implements a peer-to-peerprotocol in accordance with the bit torrent protocol standard.Information describing the bit torrent standard can be found athttp://www.bittorrent.org/beps/bep_0003.html.

FIG. 13 shows an illustrative example of a storage service used forretaining protected cryptographic material maintained by a fleet ofHSMs, in accordance with an embodiment. A diagram 1300 shows an HSMfleet that uses a storage service for backing up cryptographic keysmaintained by the fleet of HSMs. An HSM management hub 1302 includes anHSM fleet manager 1304 that maintains a fleet information database 1306.A first HSM 1308 and a second HSM 1310 comprise an HSM fleet managed bythe HSM management hub 1302. The first HSM 1308 and the second HSM 1310maintain copies of a fleet key 1312 and 1314 respectively. The first HSM1308 and the second HSM 1310 maintain key stores 1316 and 1318,respectively. The key stores retain cryptographic information includingencryption keys used by the HSMs in the fleet. The storage service 1320is in communication with the HSM management hub 1302, and maintains anencrypted copy of the key store 1322 that is encrypted with the fleetkey. The storage service 1320 is not an HSM, and may not use the keysmaintained in the encrypted copy of the key store 1322 because thestorage service 1320 does not have access to the fleet key. Bymaintaining an encrypted copy of the key store, the storage service 1320enables backup and recovery services for the HSM fleet.

Updated cryptographic information including encryption keys isreplicated across the fleet of HSMs and the storage service 1320. In oneexample, the first HSM 1308 generates an updated cryptographic key whichis stored in the key store 1316. The updated key store is encryptedusing the fleet key 1312 and transmitted to the HSM management hub 1302.The HSM management hub 1302 transmits the updated key store to thesecond HSM 1310 and storage service 1320. The second HSM 1310 decryptsthe updated key store with a copy of the fleet key 1314, and reconcilesthe updated cryptographic information in the updated key store with thecryptographic information maintained in the key store 1318 on the secondHSM 1310. The storage service 1320 receives the updated key store fromthe HSM management hub 1302, but cannot decrypt the updated key storebecause the storage service 1320 does not have access to the fleet key.Instead, the storage service 1320 retains the updated key store as theencrypted copy of the key store 1322.

If an HSM within the HSM fleet fails, information maintained by thestorage service 1320 may be used to recover the cryptographicinformation lost by the failing HSM. In some examples, if an HSM in theHSM fleet fails, the failed HSM can be replaced by new HSM. The new HSMjoins the fleet and is provided with the fleet key by another HSM in thefleet as described elsewhere in the present document. In someimplementations the new HSM acquires the fleet key by restoring thefleet key from physical media. The storage service 1320 is contacted bythe new HSM, and the storage service 1320 provides the new HSM with themost recent encrypted copy of the key store 1322. In someimplementations, the HSM management hub 1302 detects the new HSM, andcontacts the storage service 1320 to retrieve the most recent encryptedcopy of the key store 1322. The HSM management hub 1302 ensures the newHSM is a member of the fleet, and provides the encrypted copy of the keystore to the new HSM. In another implementation, the new HSM joins thefleet of HSMs, and then broadcasts a message to the fleet of HSMsrequesting restoration of the key store. The storage service 1320 mayrespond to the broadcast message by providing the encrypted key store,or a current member of the HSM fleet may respond by encrypting itspresent key store with the current fleet key and providing it to the newHSM. The new HSM decrypts the encrypted copy of the key store with thefleet key, and restores the key store on the new HSM.

FIG. 14 shows an illustrative example of cryptographic materialdistributed across a fleet of customer HSMs, with associated parityinformation maintained by an HSM management hub, in accordance with anembodiment. A diagram 1400 shows a fleet of HSMs being managed by an HSMmanagement hub 1402. The HSM management hub 1402 manages three HSMs: afirst HSM 1404, a second HSM 1406, and a third HSM 1408. Each HSM in thefleet of HSMs maintains a copy of a fleet key 1410, 1412, and 1414. Invarious embodiments, the fleet of HSMs can maintain a backup ofcryptographic information that is distributed across to a number of HSMsin the HSM fleet. In the example shown in FIG. 14, the first HSM 1404retains a first backup key stripe 1416, the second HSM 1406 retains asecond backup key stripe 1418, and the third HSM 1408 retains a thirdbackup key stripe 1420. If the stripes are combined, they produce acomplete backup of the cryptographic information retained on the HSMfleet. The HSM management hub 1402 includes an HSM fleet manager 1422that maintains a backup key parity volume 1424. The backup key parityvolume 1424 includes parity information that allows for the detectionand, if necessary, recovery of one or more backup key stripes. In oneexample, the backup key parity volume 1424 includes a parity bit thatallows recovery from the loss of a single backup key stripe. Theinformation stored in the backup key parity volume 1424 may be designedto allow recovery from the loss of one, two, or more HSMs from the HSMfleet. The parity information described herein may be replaced withother types of error correcting codes such as Reed Solomon codes orerasure codes. Additional information on the design and implementationof error correcting codes may be found in “Error Correcting Codes”2^(nd) edition by Peterson and Weldon, copyright 1972 by MIT. In variousimplementations, the cryptographic information may be divided intosubsets, overlapping portions, or segments which may be treated asstripes for the purpose of generating parity information or errorcorrecting codes.

The backup key parity volume 1424 may be maintained without revealingthe cryptographic information maintained in the HSMs to the HSMmanagement hub 1402. In one example, updated cryptographic informationis generated on the first HSM 1404. The first HSM 1404 generates abackup volume containing the cryptographic information, and divides thevolume into three stripes, and parity information. The first stripe isretained by the first HSM 1404. The second stripe is distributed to thesecond HSM 1406, and the third stripe is distributed to the third HSM1408. The parity information is transmitted to the HSM management hub1402 which stores the parity information in the backup key parity volume1424.

If an HSM within the HSM fleet fails, the HSM management hub providesthe parity information to a designated HSM in the fleet. The designatedHSM assembles the backup stripes from the other HSMs in the fleet andcombines this information with the parity information provided by theHSM management hub 1402 to reconstruct the missing backup stripe. Thedesignated HSM restores the backup information, which may be used torestore the failed HSM or, in some implementations, may be used toredistribute the backup information across the remaining HSMs. In thisway, cryptographic information maintained on an HSM fleet may be backedup using a relatively small amount of storage on each HSM. In addition,parity information may be hosted by storage services or management hubsthat do not necessarily have access to the cryptographic information onthe HSMs. The parity information allows for the recovery of lostcryptographic information on the HSMs without entrusting thecryptographic information to a third-party backup service.

The parity information can be used to achieve a level of data storageredundancy. If cryptographic information stored on an HSM becomescorrupted or is lost, the information may be recovered by combining thedata on the remaining HSMs with parity information. In someimplementations, the parity information is combined with a Booleanexclusive or (“XOR”) operation to recover the lost information.

For example, if two HSMs contained the following data:

HSM 1: 01101101

HSM 2: 11010100

The parity information for the two HSMs can be determined by performingan XOR operation on the data. In the above example, the parityinformation would be:

HSM 1: 01101101

HSM 2: 11010100

Parity: 10111001

The resulting parity data, 10111001, can be stored on an HSM, on an HSMmanagement hub, or storage service. If either HSM1 or HSM2 fails, theinformation stored on the failed HSM can be recovered from theinformation remaining on the other HSM by combining the remaininginformation with the parity information with an XOR operation.

For example, if HSM 2 failed, the data could be recovered by performingan XOR on the remaining data and the parity information.

HSM 1: 01101101

Parity: 10111001

Recovered HSM 2: 11010100

The result of that XOR calculation yields HSM 2's contents, 11010100.The recovery algorithm may be applied to larger numbers of HSMs. Parityinformation may be generated for 3, 4, or more HSMs and be used torecover from any single failure.

FIG. 15 shows an illustrative example of a storage service used toretain parity information for cryptographic information maintained by afleet of HSMs, in accordance with an embodiment. An environment 1500includes an HSM management hub 1502 that manages a fleet of three HSMs;a first HSM 1504, a second HSM 1506, and a third HSM 1508. The HSMmanagement hub includes an HSM fleet backup manager 1510. The HSM fleetbackup manager 1510 is an operational component that manages the backupand recovery of cryptographic information maintained by the HSM fleet.Each HSM in the HSM fleet maintains a copy of the fleet key 1512, 1514,and 1516. A backup volume is generated by a designated HSM, and dividedinto three backup key stripes. The first HSM 1504 retains a first backupkey stripe 1518. The second HSM 1506 retains a second backup key stripe1520, and the third HSM 1508 retains a third backup key stripe 1522.Parity information is provided by the designated HSM to the HSM fleetbackup manager 1510. The HSM fleet backup manager 1510 utilizes one ormore storage services 1524. The storage service 1524 retains a backupkey stripe parity volume 1526, the contents of which are based at leastin part on the parity information provided by the designated HSM. Thebackup information provided by the designated HSM is encrypted using thefleet key, and is provided to the storage service 1524 in encryptedform. The HSM fleet backup manager 1510 may utilize additional storageservices to store additional parity information provided by thedesignated HSM. Dividing parity information across multiple storageservice providers can provide additional security by making it moredifficult for an attacker to acquire a significant portion of thecryptographic information maintained on the HSM fleet.

If an HSM in the HSM fleet fails, the HSM fleet backup manager 1510retrieves the parity information from the storage service 1524. Areplacement HSM is added to the HSM fleet and the fleet key isreplicated from an existing HSM in the fleet to the replacement HSM. TheHSM fleet backup manager 1510 provides the parity information to adesignated HSM in the fleet. The designated HSM retrieves the remainingbackup key stripes from the other HSMs in the fleet, and using theparity information, regenerates the missing stripe that was lost whenthe HSM failed. The missing stripe is provided to the new HSM. In someimplementations, the parity information is provided to the replacementHSM. The replacement HSM retrieves the remaining backup key stripes fromthe other HSMs in the fleet, and is able to regenerate a complete backupvolume using the parity information provided by the HSM fleet backupmanager 1510, in addition to regenerating the missing backup key stripe.

In some examples, the HSM fleet backup manager 1510 detects that an HSMin the fleet of HSMs has failed, resulting in the loss of a backup keystripe. The HSM fleet backup manager 1510 identifies the location of theremaining stripes, and the parity information and assembles theremaining stripes and parity information within the HSM management hub1502. The HSM management hub identifies the replacement HSM, andconfirms that the replacement HSM has been provided with a copy of thefleet key. The HSM management hub provides the remaining stripes and theparity information to the replacement HSM, and the replacement HSM usesthe fleet key to decrypt the remaining stripes and the parityinformation. The decoded stripes and parity information are used toregenerate the missing stripe, and restore the cryptographic informationon the replacement HSM.

FIG. 16 shows an illustrative example of a process that, as a result ofbeing performed by an HSM fleet and a storage service, backs upinformation stored on the HSM fleet by exporting parity information tothe storage service, in accordance with an embodiment. A swim diagram1600 shows a process that begins at block 1602 with a particular HSMwithin an HSM fleet updating cryptographic information maintained by theparticular HSM. As a result of the cryptographic information beingupdated, the particular HSM generates 1604 a volume of backupinformation that represents the cryptographic information maintained bythe particular HSM. The backup information may be encoded with the fleetkey, an HSM manufacturer's key, or designated backup key to secure thebackup information. At block 1606, the particular HSM divides the backupvolume into a number of stripes. The particular HSM generates 1608parity information that can be used to regenerate one or more slices inthe event that a particular stripe is lost or corrupted. In oneimplementation, the backup volume is divided into a number of stripes ofequal size, and parity information is a set of exclusive or parity bitsfor the backup stripes. At block 1610, the particular HSM distributesthe backup stripes to fleet-member HSMs so that each HSM in the HSMfleet maintains a different stripe. The parity information is uploaded1612 to a storage service. The parity information may be encrypted usingthe fleet key, a backup key, or other encryption key. At block 1614, thestorage service receives the parity information from the particular HSMand stores the parity information.

FIG. 17 shows an illustrative example of a process that, as a result ofbeing performed by an HSM fleet and a storage service, restores a failedHSM using parity information retained by the storage service, inaccordance with an embodiment. A swim diagram 1700 illustrates a processthat begins at block 1702 with a designated HSM determining that one ormore HSMs in HSM fleet has failed. In order to recover from the HSMfailure, the designated HSM requests 1704 parity information from thestorage service.

The storage service receives the request for parity information from thedesignated HSM. At block 1706, the storage service authenticates therequest by confirming that the requesting HSM is a member of the HSMfleet, and that the requesting HSM is authorized to receive the parityinformation. At block 1708, the storage service provides the parityinformation to the designated HSM.

The designated HSM receives 1710 the parity information from the storageservice and assembles the remaining backup volume stripes from theremaining HSMs of the HSM fleet. At block 1712, the designated HSMgenerates the missing backup stripe or stripes using the parityinformation and the surviving backup volume stripes. In someimplementations, the parity information provided by the storage serviceallows for the recovery of a single backup stripe. In otherimplementations, the parity information provided by the storage serviceallows for the recovery of multiple backup stripes by increasing theamount of parity information stored on the storage service. At block1714, the failed HSM is replaced with a replacement HSM, and thereplacement HSM is integrated into the HSM fleet. As part of integratingthe replacement HSM, the replacement HSM is provided with a fleet key,backup key, or other cryptographic key necessary to decrypt the backupinformation. At block 1716, the designated HSM distributes the restoredbackup information to the replacement HSM. The replacement HSM decryptsthe backup information if necessary, and installs the cryptographicinformation contained in the backup information onto the replacementHSM.

FIG. 18 illustrates aspects of an example environment 1800 forimplementing aspects in accordance with various embodiments. As will beappreciated, although a web-based environment is used for purposes ofexplanation, different environments may be used, as appropriate, toimplement various embodiments. The environment includes an electronicclient device 1802, which can include any appropriate device operable tosend and/or receive requests, messages, or information over anappropriate network 1804 and, in some embodiments, convey informationback to a user of the device. Examples of such client devices includepersonal computers, cell phones, handheld messaging devices, laptopcomputers, tablet computers, set-top boxes, personal data assistants,embedded computer systems, electronic book readers, and the like. Thenetwork can include any appropriate network, including an intranet, theInternet, a cellular network, a local area network, a satellite networkor any other such network and/or combination thereof. Components usedfor such a system can depend at least in part on the type of networkand/or environment selected. Protocols and components for communicatingvia such a network are well known and will not be discussed herein indetail. Communication over the network can be enabled by wired orwireless connections and combinations thereof. In this example, thenetwork includes the Internet, as the environment includes a web server1806 for receiving requests and serving content in response thereto,although for other networks an alternative device serving a similarpurpose could be used as would be apparent to one of ordinary skill inthe art.

The illustrative environment includes at least one application server1808 and a data store 1810. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. Servers, as used herein, may be implemented in various ways,such as hardware devices or virtual computer systems. In some contexts,servers may refer to a programming module being executed on a computersystem. As used herein, unless otherwise stated or clear from context,the term “data store” refers to any device or combination of devicescapable of storing, accessing and retrieving data, which may include anycombination and number of data servers, databases, data storage devicesand data storage media, in any standard, distributed, virtual orclustered environment. The application server can include anyappropriate hardware, software and firmware for integrating with thedata store as needed to execute aspects of one or more applications forthe client device, handling some or all of the data access and businesslogic for an application. The application server may provide accesscontrol services in cooperation with the data store and is able togenerate content including, but not limited to, text, graphics, audio,video and/or other content usable to be provided to the user, which maybe served to the user by the web server in the form of HyperText MarkupLanguage (“HTML”), Extensible Markup Language (“XML”), JavaScript,Cascading Style Sheets (“CSS”), or another appropriate client-sidestructured language. Content transferred to a client device may beprocessed by the client device to provide the content in one or moreforms including, but not limited to, forms that are perceptible to theuser audibly, visually and/or through other senses including touch,taste, and/or smell. The handling of all requests and responses, as wellas the delivery of content between the client device 1802 and theapplication server 1808, can be handled by the web server using PHP:Hypertext Preprocessor (“PHP”), Python, Ruby, Perl, Java, HTML, XML, oranother appropriate server-side structured language in this example. Itshould be understood that the web and application servers are notrequired and are merely example components, as structured code discussedherein can be executed on any appropriate device or host machine asdiscussed elsewhere herein. Further, operations described herein asbeing performed by a single device may, unless otherwise clear fromcontext, be performed collectively by multiple devices, which may form adistributed and/or virtual system.

The data store 1810 can include several separate data tables, databases,data documents, dynamic data storage schemes and/or other data storagemechanisms and media for storing data relating to a particular aspect ofthe present disclosure. For example, the data store illustrated mayinclude mechanisms for storing production data 1812 and user information1816, which can be used to serve content for the production side. Thedata store also is shown to include a mechanism for storing log data1814, which can be used for reporting, analysis or other such purposes.It should be understood that there can be many other aspects that mayneed to be stored in the data store, such as page image information andaccess rights information, which can be stored in any of the abovelisted mechanisms as appropriate or in additional mechanisms in the datastore 1810. The data store 1810 is operable, through logic associatedtherewith, to receive instructions from the application server 1808 andobtain, update or otherwise process data in response thereto. Theapplication server 1808 may provide static, dynamic, or a combination ofstatic and dynamic data in response to the received instructions.Dynamic data, such as data used in web logs (blogs), shoppingapplications, news services and other such applications may be generatedby server-side structured languages as described herein or may beprovided by a content management system (“CMS”) operating on, or underthe control of, the application server. In one example, a user, througha device operated by the user, might submit a search request for acertain type of item. In this case, the data store might access the userinformation to verify the identity of the user and can access thecatalog detail information to obtain information about items of thattype. The information then can be returned to the user, such as in aresults listing on a web page that the user is able to view via abrowser on the client device 1802. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser. Itshould be noted, however, that embodiments of the present disclosure arenot necessarily limited to the context of web pages, but may be moregenerally applicable to processing requests in general, where therequests are not necessarily requests for content.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include a computer-readablestorage medium (e.g., a hard disk, random access memory, read onlymemory, etc.) storing instructions that, when executed by a processor ofthe server, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment, in one embodiment, is a distributed and/or virtualcomputing environment utilizing several computer systems and componentsthat are interconnected via communication links, using one or morecomputer networks or direct connections. However, it will be appreciatedby those of ordinary skill in the art that such a system could operateequally well in a system having fewer or a greater number of componentsthan are illustrated in FIG. 18. Thus, the depiction of the system 1800in FIG. 18 should be taken as being illustrative in nature and notlimiting to the scope of the disclosure.

The various embodiments further can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers, computing devices or processing devices which can be used tooperate any of a number of applications. User or client devices caninclude any of a number of general purpose personal computers, such asdesktop, laptop or tablet computers running a standard operating system,as well as cellular, wireless and handheld devices running mobilesoftware and capable of supporting a number of networking and messagingprotocols. Such a system also can include a number of workstationsrunning any of a variety of commercially-available operating systems andother known applications for purposes such as development and databasemanagement. These devices also can include other electronic devices,such as dummy terminals, thin-clients, gaming systems and other devicescapable of communicating via a network. These devices also can includevirtual devices such as virtual machines, hypervisors and other virtualdevices capable of communicating via a network.

Various embodiments of the present disclosure utilize at least onenetwork that would be familiar to those skilled in the art forsupporting communications using any of a variety ofcommercially-available protocols, such as Transmission ControlProtocol/Internet Protocol (“TCP/IP”), User Datagram Protocol (“UDP”),protocols operating in various layers of the Open System Interconnection(“OSI”) model, File Transfer Protocol (“FTP”), Universal Plug and Play(“UpnP”), Network File System (“NFS”), Common Internet File System(“CIFS”), and AppleTalk. The network can be, for example, a local areanetwork, a wide-area network, a virtual private network, the Internet,an intranet, an extranet, a public switched telephone network, aninfrared network, a wireless network, a satellite network, and anycombination thereof. In some embodiments, connection-oriented protocolsmay be used to communicate between network endpoints.Connection-oriented protocols (sometimes called connection-basedprotocols) are capable of transmitting data in an ordered stream.Connection-oriented protocols can be reliable or unreliable. Forexample, the TCP protocol is a reliable connection-oriented protocol.Asynchronous Transfer Mode (“ATM”) and Frame Relay are unreliableconnection-oriented protocols. Connection-oriented protocols are incontrast to packet-oriented protocols such as UDP that transmit packetswithout a guaranteed ordering.

In embodiments utilizing a web server, the web server can run any of avariety of server or mid-tier applications, including Hypertext TransferProtocol (“HTTP”) servers, FTP servers, Common Gateway Interface (“CGP”)servers, data servers, Java servers, Apache servers, and businessapplication servers. The server(s) also may be capable of executingprograms or scripts in response to requests from user devices, such asby executing one or more web applications that may be implemented as oneor more scripts or programs written in any programming language, such asJava®, C, C#, or C++, or any scripting language, such as Ruby, PHP,Perl, Python or TCL, as well as combinations thereof. The server(s) mayalso include database servers, including without limitation thosecommercially available from Oracle®, Microsoft®, Sybase and IBM® as wellas open-source servers such as MySQL, Postgres, SQLite, MongoDB, and anyother server capable of storing, retrieving, and accessing structured orunstructured data. Database servers may include table-based servers,document-based servers, unstructured servers, relational servers,non-relational servers or combinations of these and/or other databaseservers.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (“CPU” or “processor”), atleast one input device (e.g., a mouse, keyboard, controller, touchscreen or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (“RAM”) orread-only memory (“ROM”), as well as removable media devices, memorycards, flash cards, etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device, etc.), and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed, and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as, but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, Electrically ErasableProgrammable Read-Only Memory (“EEPROM”), flash memory or other memorytechnology, Compact Disc Read-Only Memory (“CD-ROM”), digital versatiledisk (DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices or any othermedium which can be used to store the desired information and which canbe accessed by the system device. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will appreciateother ways and/or methods to implement the various embodiments.

As discussed, numerous variations utilize symmetric and/or asymmetriccryptographic primitives. Symmetric key algorithms may include variousschemes for performing cryptographic operations on data including blockciphers, stream ciphers and digital signature schemes. Example symmetrickey algorithms include the advanced encryption standard (AES), the dataencryption standard (DES), triple DES (3DES), Serpent, Twofish,blowfish, CASTS, RC4 and the international data encryption algorithm(IDEA). Symmetric key algorithms may also include those used to generateoutput of one way functions and include algorithms that utilizehash-based message authentication codes (HMACs), message authenticationcodes (MACs) in general, PBKDF2, and Bcrypt. Asymmetric key algorithmsmay also include various schemes for performing cryptographic operationson data. Example algorithms include those that utilize theDiffie-Hellman key exchange protocol, the digital signature standard(DSS), the digital signature algorithm, the ElGamal algorithm, variouselliptic curve algorithms, password-authenticated key agreementtechniques, the pallier cryptosystem, the RSA encryption algorithm(PKCS#1), the Cramer-Shoup cryptosystem, the YAK authenticated keyagreement protocol, the NTRUEncrypt cryptosystem, the McEliececryptosystem, and others. Elliptic curve algorithms include the ellipticcurve Diffie-Hellman (ECDH) key agreement scheme, the Elliptic CurveIntegrated Encryption Scheme (ECIES), the Elliptic Curve DigitalSignature Algorithm (ECDSA), the ECMQV key agreement scheme, and theECQV implicit certificate scheme. Other algorithms and combinations ofalgorithms are also considered as being within the scope of the presentdisclosure and the above is not intended to be an exhaustive list.

Note that the term “digital signature” includes any information usableto cryptographically verify authenticity of a message includinginformation generated using an RSA-based digital scheme (such asRSA-PSS), the digital signature algorithm (DSA) and the elliptic curvedigital signature algorithm, the ElGamal signature scheme, the Schnorrsignature scheme, the Pointcheval-Stern signature algorithm, the Rabinsignature algorithm, pairing-based digital signature schemes (such asthe Boneh-Lynn-Schacham signature scheme), undeniable digital signatureschemes, and others. Further, message authentication codes (such ashash-based message authentication codes (HMACs), keyed cryptographichash functions, and other types of information may also be used asdigital signatures.

Generally, embodiments of the present disclosure may use variousprotocols, such as a SSL or TLS protocol and extensions thereto, such asdefined in Request for Comments (RFC) 2246, RFC 2595, RFC 2712, RFC2817, RFC 2818, RFC 3207, RFC 3268, RFC 3546, RFC 3749, RFC 3943, RFC4132, RFC 4162, RFC 4217, RFC 4279, RFC 4347, RFC 4366, RFC 4492, RFC4680, RFC 4681, RFC 4785, RFC 5054, RFC 5077, RFC 5081, RFC 5238, RFC5246, RFC 5288, RFC 5289, RFC 5746, RFC 5764, RFC 5878, RFC 5932, RFC6083, RFC 6066, RFC 6091, RFC 6176, RFC 6209, RFC 6347, RFC 6367, RFC6460, RFC 6655, RFC 7027, and RFC 7366, which are incorporated herein byreference, to establish encrypted communications sessions. Otherprotocols implemented below the application layer of the Open SystemsInterconnect (OSI) model may also be used and/or adapted to utilizetechniques described herein. It should be noted that the techniquesdescribed herein are adaptable to other protocols such as the Real TimeMessaging Protocol (RTMP), the Point-to-Point Tunneling Protocol (PPTP),the Layer 2 Tunneling Protocol, various virtual private network (VPN)protocols, Internet Protocol Security (e.g., as defined in RFC 1825through 1829, RFC 2401, RFC 2412, RFC 4301, RFC 4309, and RFC 4303) andother protocols, such as protocols for secure communication that includea handshake. Note also that the examples used herein may be performed incompliance with one or more of: Request for Comments (RFC) 4250, RFC4251, RFC 4252, RFC 4253, RFC 4254, RFC 4255, RFC 4256, RFC 4335, RFC4344, RFC 4345, RFC 4419, RFC 4432, RFC 4462, RFC 4716, RFC 4819, RFC5647, RFC 5656, RFC 6187, RFC 6239, RFC 6594, and RFC 6668, which areincorporated by reference.

Note that a system is said to be configured to trust a publiccryptographic key if logic with which the system is configured tooperate is dependent on whether an attempt to verify a digital signaturewith the public cryptographic key is successful. Similarly, a system issaid to be configured to trust a symmetric cryptographic key if logicwith which the system is configured to operate is dependent on whetheran attempt to verify a digital signature with the symmetriccryptographic key is successful.

In various embodiments, data objects such as digital signatures may becryptographically verifiable. In one example, cryptographicallyverifiable data objects are created to be cryptographically verifiableby the system to which the data object is to be provided or anothersystem that operates in conjunction with the system to which the dataobject is to be provided. For example, the data object may be encryptedso as to be decryptable by the system that will cryptographically verifythe data object, where the ability to decrypt the data object serves ascryptographic verification of the data object. As another example, thedata object may be digitally signed (thereby producing a digitalsignature of the data object) such that the digital signature isverifiable by the system that will cryptographically verify the dataobject. In other examples, both encryption and digital signatures areused for cryptographic verifiability and/or security. The key used toencrypt and/or digitally sign the data object may vary in accordancewith various embodiments and the same key is not necessarily used forboth encryption and digital signing, where applicable. In someembodiments, a key used to encrypt the data object is a public key of apublic/private key pair where the private key of the key pair ismaintained securely by the system to which the data object is to beprovided, thereby enabling the system to decrypt the data object usingthe private key of the key pair. Using the public key to encrypt thedata object may include generating a symmetric key, using the symmetrickey to encrypt the data object, and encrypting the symmetric key usingthe public key, where the encrypted symmetric key is provided to asystem with the encrypted data object to enable the system to use thecorresponding private key to decrypt the symmetric key and use thedecrypted symmetric key to decrypt the data object. Further, in someembodiments, the data object is digitally signed using a private key ofa public/private key pair corresponding to the computer system thatencrypts and/or digitally signs the data object (e.g., a user device).For example, an application may be provisioned with the private key andthe data object may include a certificate for the private key for use bya system for verification of the digital signature of the data object.Other variations, including variations where a symmetric key sharedbetween the user computer and the system that cryptographically verifiesthe data object can be used to encrypt and/or digitally sign the dataobject.

In the preceding and following description, various techniques aredescribed. For purposes of explanation, specific configurations anddetails are set forth in order to provide a thorough understanding ofpossible ways of implementing the techniques. However, it will also beapparent that the techniques described in this document may be practicedin different configurations without the specific details. Furthermore,well-known features may be omitted or simplified to avoid obscuring thetechniques being described.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructionsand equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected,” when unmodified and referring to physical connections, isto be construed as partly or wholly contained within, attached to orjoined together, even if there is something intervening. Recitation ofranges of values herein are merely intended to serve as a shorthandmethod of referring individually to each separate value falling withinthe range, unless otherwise indicated herein and each separate value isincorporated into the specification as if it were individually recitedherein. The use of the term “set” (e.g., “a set of items”) or “subset”unless otherwise noted or contradicted by context, is to be construed asa nonempty collection comprising one or more members. Further, unlessotherwise noted or contradicted by context, the term “subset” of acorresponding set does not necessarily denote a proper subset of thecorresponding set, but the subset and the corresponding set may beequal.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. Further, in some examples, computersystems are configured to implement one or more services thatcollectively perform operations of processes described herein. Suchcomputer systems may, for instance, be configured with applicablehardware and/or software that enable the performance of the operations.Further, computer systems that implement various embodiments of thepresent disclosure may, in some examples, be single devices and, inother examples, be distributed computer systems comprising multipledevices that operate differently such that the distributed computersystem performs the operations described herein.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

Embodiments of this disclosure are described herein, including the bestmode known to the inventors for carrying out the invention. Variationsof those embodiments may become apparent to those of ordinary skill inthe art upon reading the foregoing description. The inventors expectskilled artisans to employ such variations as appropriate and theinventors intend for embodiments of the present disclosure to bepracticed otherwise than as specifically described herein. Accordingly,the scope of the present disclosure includes all modifications andequivalents of the subject matter recited in the claims appended heretoas permitted by applicable law. Moreover, any combination of theabove-described elements in all possible variations thereof isencompassed by the scope of the present disclosure unless otherwiseindicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A computer-implemented method comprising:storing, in a hardware security module, a fleet key that is replicatedacross a fleet of hardware security modules, and a domain key that isreplicated across a subset of hardware security modules in the fleet ofhardware security modules; receiving cryptographic material in anencrypted format, the cryptographic material encrypted with the fleetkey, a first portion of the cryptographic material encrypted with thedomain key; decrypting the first portion of the cryptographic materialwith at least the fleet key and the domain key; and distributing updatedcryptographic material that is cryptographically protected with thefleet key to an additional hardware security module in the fleet ofhardware security modules.
 2. The computer-implemented method of claim1, wherein: the additional hardware security module belongs to thesubset of hardware security modules; and the updated cryptographicmaterial is encrypted with at least the domain key.
 3. Thecomputer-implemented method of claim 1 further comprising: distributingthe updated cryptographic material to another hardware security modulethat is not in the subset of hardware security modules, and that doesnot have a copy of the domain key; and causing the other hardwaresecurity module to store the updated cryptographic material in encryptedform.
 4. The computer-implemented method of claim 1 further comprising:the additional hardware security module is an internal hardware securitymodule hosted by a host computer system; and distributing the updatedcryptographic material to the additional hardware security module isaccomplished at least by relaying the updated cryptographic materialthrough a driver running on the host computer system.
 5. A system,comprising: one or more processors; and memory storingcomputer-executable instructions that, if executed, cause the system to:synchronize cryptographic material between hardware security modulesthat are arranged in a fleet of hardware security modules at least by:retaining a fleet key that is replicated across the fleet of hardwaresecurity modules, and a first domain key that is replicated across asubset of hardware security modules in the fleet of hardware securitymodules; receiving the cryptographic material in an encrypted format,the cryptographic material encrypted with the fleet key, a first portionof the cryptographic material encrypted with the first domain key;decrypting the cryptographic material with the fleet key and the firstdomain key; and distributing updated cryptographic material that iscryptographically protected with the fleet key to at least oneadditional hardware security module in the fleet of hardware securitymodules.
 6. The system of claim 5, wherein distributing the updatedcryptographic material is accomplished at least in part by providing theupdated cryptographic material to an HSM management hub that is incommunication with the fleet of hardware security modules.
 7. The systemof claim 5, wherein distributing the updated cryptographic material isaccomplished at least in part by sending, via a peer-to-peer networkingprotocol, the updated cryptographic material to at least onefleet-member hardware security module within the fleet of hardwaresecurity modules.
 8. The system of claim 5, wherein the system furthermerges the cryptographic material with existing cryptographic materialstored on the at least one hardware security module.
 9. The system ofclaim 5, wherein the fleet key is a public-private key pair and theupdated cryptographic material is protected in part by encrypting theupdated cryptographic material with a public key of the public-privatekey pair.
 10. The system of claim 5, wherein the fleet key is asymmetric cryptographic key and the updated cryptographic material isprotected in part by encrypting the updated cryptographic material withthe fleet key.
 11. The system of claim 5, wherein the system furthersynchronizes cryptographic material across the fleet of hardwaresecurity modules at least by: digitally signing the updatedcryptographic material using a cryptographic key associated with the atleast one hardware security module; and verifying a digital signature onthe cryptographic material as a condition of accessing the cryptographicmaterial.
 12. The system of claim 5, wherein the system furthersynchronizes cryptographic material across the fleet of hardwaresecurity modules at least by: generating an updated cryptographic key asa replacement for an original cryptographic key; cryptographicallyprotecting the updated cryptographic key with the original cryptographickey and the fleet key; and causing the updated cryptographic key to bedistributed across the fleet of hardware security modules.
 13. Anon-transitory computer-readable storage medium having stored thereonexecutable instructions that, as a result of being executed by one ormore processors of a hardware security module, cause the hardwaresecurity module to at least: retain a fleet key that is replicatedacross a fleet of hardware security modules, and a domain key that isreplicated across a set of hardware security modules in the fleet ofhardware security modules; receive cryptographic material in anencrypted format, the cryptographic material encrypted with the fleetkey, a first portion of the cryptographic material encrypted with thedomain key; decrypt the cryptographic material with the fleet key andthe domain key to produce a cryptographic key; and perform acryptographic operation using the cryptographic key.
 14. Thenon-transitory computer-readable storage medium of claim 13, wherein theinstructions further comprise instructions that, as a result of beingexecuted by the one or more processors, cause the hardware securitymodule to: generate a new cryptographic key; encrypt the newcryptographic key with the domain key and the fleet key to produceencrypted updated information; and cause the encrypted updatedinformation to be distributed to the set of hardware security modules inthe fleet of hardware security modules.
 15. The non-transitorycomputer-readable storage medium of claim 13, wherein the set ofhardware security modules is selected from the fleet of hardwaresecurity modules based at least in part on fleet-configurationinformation maintained in a fleet information store.
 16. Thenon-transitory computer-readable storage medium of claim 13, wherein theinstructions further comprise instructions that, as a result of beingexecuted by the one or more processors, causes the hardware securitymodule to: receive, from a domain-lead hardware security module in theset of hardware security modules, an encrypted updated domain key thatis encrypted with at least the fleet key; decrypt the encrypted updateddomain key with the fleet key to obtain an updated domain key; and causeencrypted information to be distributed to the set of hardware securitymodules, the encrypted information encrypted using the updated domainkey.
 17. The non-transitory computer-readable storage medium of claim13, wherein the instructions further comprise instructions that, as aresult of being executed by the one or more processors, causes thehardware security module to: receive, from a domain-lead hardwaresecurity module in the set of hardware security modules, an encryptedupdated domain key that is encrypted with at least the domain key;decrypt the encrypted updated domain key with the domain key to obtainan updated domain key; and receive encrypted information from a sourcehardware security module in the set of hardware security modules, theencrypted information encrypted using the updated domain key.
 18. Thenon-transitory computer-readable storage medium of claim 13, wherein theinstructions further comprise instructions that, as a result of beingexecuted by the one or more processors, causes the hardware securitymodule to: receive, from a requesting computer system, a request forupdated cryptographic information; determine whether updatedcryptographic material is available; encrypt the updated cryptographicmaterial with the fleet key to produce encrypted cryptographic material;and provide the encrypted cryptographic material to the requester. 19.The non-transitory computer-readable storage medium of claim 13, whereinthe instructions further comprise instructions that, as a result ofbeing executed by the one or more processors, causes the hardwaresecurity module to: determine that cryptographic information stored onthe hardware security module has changed; and provide, to a hardwaresecurity module management hub, a notification that the hardwaresecurity module includes updated cryptographic information.
 20. Thenon-transitory computer-readable storage medium of claim 13, wherein theinstructions further comprise instructions that, as a result of beingexecuted by the one or more processors, causes the hardware securitymodule to: receive, from a hardware security module management hub, anencrypted fleet management key; validate the encrypted fleet managementkey using a public key associated with a fleet administrator; decryptthe encrypted fleet management key with a hardware security modulemanufacturer's key to produce a fleet key; and join the fleet by atleast storing the fleet key on the hardware security module.